{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23R0Z9RojXYW"
      },
      "source": [
        "# Ungraded Lab: Model Analysis with TFX Evaluator"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sfSQ-kX-MLEr"
      },
      "source": [
        "Now that you've used TFMA as a standalone library in the previous lab, you will now see how it is used by TFX with its [Evaluator](https://www.tensorflow.org/tfx/guide/evaluator) component. This component comes after your `Trainer` run and it checks if your trained model meets the minimum required metrics and also compares it with previously generated models.\n",
        "\n",
        "You will go through a TFX pipeline that prepares and trains the same model architecture you used in the previous lab. As a reminder, this is a binary classifier to be trained on the [Census Income dataset](http://archive.ics.uci.edu/ml/datasets/Census+Income). Since you're already familiar with the earlier TFX components, we will just go over them quickly but we've placed notes on where you can modify code if you want to practice or produce a better result.\n",
        "\n",
        "Let's begin!\n",
        "\n",
        "*Credits: Some of the code and discussions are based on the TensorFlow team's [official tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/components_keras).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2GivNBNYjb3b"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZOYTt1RW4TK"
      },
      "source": [
        "### Install TFX"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S4SQA7Q5nej3"
      },
      "source": [
        "!pip install -U pip\n",
        "!pip install tfx==1.12.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ia3dHRt72hE8"
      },
      "source": [
        "*Note: In Google Colab, you need to restart the runtime at this point to finalize updating the packages you just installed. You can do so by clicking the `Restart Runtime` at the end of the output cell above (after installation), or by selecting `Runtime > Restart Runtime` in the Menu bar. **Please do not proceed to the next section without restarting.** You can also ignore the errors about version incompatibility of some of the bundled packages because we won't be using those in this notebook.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-ePgV0Lj68Q"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YIqpWK9efviJ"
      },
      "source": [
        "import os\n",
        "import pprint\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_model_analysis as tfma\n",
        "from tfx import v1 as tfx\n",
        "from absl import logging\n",
        "\n",
        "from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext\n",
        "\n",
        "tf.get_logger().propagate = False\n",
        "tf.get_logger().setLevel('ERROR')\n",
        "pp = pprint.PrettyPrinter()\n",
        "logging.set_verbosity(logging.ERROR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ufJKQ6OvkJlY"
      },
      "source": [
        "### Set up pipeline paths"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ad5JLpKbf6sN"
      },
      "source": [
        "# Location of the pipeline metadata store\n",
        "_pipeline_root = './pipeline/'\n",
        "\n",
        "# Directory of the raw data files\n",
        "_data_root = './data/census'\n",
        "\n",
        "_data_filepath = os.path.join(_data_root, \"data.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kO_3r87OPvaH"
      },
      "source": [
        "# Create the TFX pipeline files directory\n",
        "!mkdir {_pipeline_root}\n",
        "\n",
        "# Create the dataset directory\n",
        "!mkdir -p {_data_root}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2cMMAbSkGfX"
      },
      "source": [
        "### Download and prepare the dataset\n",
        "\n",
        "Here, you will download the training split of the [Census Income Dataset](http://archive.ics.uci.edu/ml/datasets/Census+Income). This is twice as large as the test dataset you used in the previous lab."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BywX6OUEhAqn"
      },
      "source": [
        "# Define filename and URL\n",
        "TAR_NAME = 'C3_W4_Lab_2_data.tar.gz'\n",
        "DATA_PATH = f'https://storage.googleapis.com/mlep-public/course_3/week4/{TAR_NAME}'\n",
        "\n",
        "# Download dataset\n",
        "!wget -nc {DATA_PATH}\n",
        "\n",
        "# Extract archive\n",
        "!tar xvzf {TAR_NAME}\n",
        "\n",
        "# Delete archive\n",
        "!rm {TAR_NAME}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "blZC1sIQOWfH"
      },
      "source": [
        "Take a quick look at the first few rows."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c5YPeLPFOXaD"
      },
      "source": [
        "# Preview dataset\n",
        "!head {_data_filepath}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5VNnoYj9K89w"
      },
      "source": [
        "## TFX Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ONIE_hdkPS4"
      },
      "source": [
        "### Create the InteractiveContext\n",
        "\n",
        "As usual, you will initialize the pipeline and use a local SQLite file for the metadata store."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Rh6K5sUf9dd"
      },
      "source": [
        "# Initialize InteractiveContext\n",
        "context = InteractiveContext(pipeline_root=_pipeline_root)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L9fwt9gQk3BR"
      },
      "source": [
        "### ExampleGen\n",
        "\n",
        "You will start by ingesting the data through `CsvExampleGen`. The code below uses the default 2:1 train-eval split (i.e. 33% of the data goes to eval) but feel free to modify if you want. You can review splitting techniques [here](https://www.tensorflow.org/tfx/guide/examplegen#splitting_method)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PyXjuMt8f-9u"
      },
      "source": [
        "# Run CsvExampleGen\n",
        "example_gen = tfx.components.CsvExampleGen(input_base=_data_root)\n",
        "context.run(example_gen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "880KkTAkPeUg"
      },
      "source": [
        "# Print split names and URI\n",
        "artifact = example_gen.outputs['examples'].get()[0]\n",
        "print(artifact.split_names, artifact.uri)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "csM6BFhtk5Aa"
      },
      "source": [
        "### StatisticsGen\n",
        "You will then compute the statistics so it can be used by the next components."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MAscCCYWgA-9"
      },
      "source": [
        "# Run StatisticsGen\n",
        "statistics_gen = tfx.components.StatisticsGen(\n",
        "    examples=example_gen.outputs['examples'])\n",
        "context.run(statistics_gen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HLI6cb_5WugZ"
      },
      "source": [
        "You can look at the visualizations below if you want to explore the data some more."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tLjXy7K6Tp_G"
      },
      "source": [
        "# Visualize statistics\n",
        "context.show(statistics_gen.outputs['statistics'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HLKLTO9Nk60p"
      },
      "source": [
        "### SchemaGen\n",
        "\n",
        "You can then infer the dataset schema with [SchemaGen](https://www.tensorflow.org/tfx/guide/schemagen). This will be used to validate incoming data to ensure that it is formatted correctly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ygQvZ6hsiQ_J"
      },
      "source": [
        "# Run SchemaGen\n",
        "schema_gen = tfx.components.SchemaGen(\n",
        "    statistics=statistics_gen.outputs['statistics'])\n",
        "context.run(schema_gen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zi6TxTUKXM6b"
      },
      "source": [
        "For simplicity, you will just accept the inferred schema but feel free to modify with the [TFDV API](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv) if you want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ec9vqDXpXeMb"
      },
      "source": [
        "# Visualize the inferred Schema\n",
        "context.show(schema_gen.outputs['schema'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V1qcUuO9k9f8"
      },
      "source": [
        "### ExampleValidator\n",
        "\n",
        "Next, run `ExampleValidator` to check if there are anomalies in the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XRlRUuGgiXks"
      },
      "source": [
        "# Run ExampleValidator\n",
        "example_validator = tfx.components.ExampleValidator(\n",
        "    statistics=statistics_gen.outputs['statistics'],\n",
        "    schema=schema_gen.outputs['schema'])\n",
        "context.run(example_validator)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wysHlJER_aWS"
      },
      "source": [
        "If you just used the inferred schema, then there should not be any anomalies detected. If you modified the schema, then there might be some results here and you can again use TFDV to modify or relax constraints as needed. \n",
        "\n",
        "In actual deployments, this component will also help you understand how your data evolves over time and identify data errors. For example, the first batches of data that you get from your users might conform to the schema but it might not be the case after several months. This component will detect that and let you know that your model might need to be updated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TDyAAozQcrk3"
      },
      "source": [
        "# Check results\n",
        "context.show(example_validator.outputs['anomalies'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPViEz5RlA36"
      },
      "source": [
        "### Transform\n",
        "\n",
        "Now you will perform feature engineering on the training data. As shown when you previewed the CSV earlier, the data is still in raw format and cannot be consumed by the model just yet. The transform code in the following cells will take care of scaling your numeric features and one-hot encoding your categorical variables.\n",
        "\n",
        "*Note: If you're running this exercise for the first time, we advise that you leave the transformation code as is. After you've gone through the entire notebook, then you can modify these for practice and see what results you get. Just make sure that your model builder code in the `Trainer` component will also reflect those changes if needed. For example, removing a feature here should also remove an input layer for that feature in the model.*\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PuNSiUKb4YJf"
      },
      "source": [
        "# Set the constants module filename\n",
        "_census_constants_module_file = 'census_constants.py'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HPjhXuIF4YJh"
      },
      "source": [
        "%%writefile {_census_constants_module_file}\n",
        "\n",
        "# Features with string data types that will be converted to indices\n",
        "VOCAB_FEATURE_DICT = {\n",
        "    'education': 16, 'marital-status': 7, 'occupation': 15, 'race': 5, \n",
        "    'relationship': 6, 'workclass': 9, 'sex': 2, 'native-country': 42\n",
        "}\n",
        "\n",
        "# Numerical features that are marked as continuous\n",
        "NUMERIC_FEATURE_KEYS = ['fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']\n",
        "\n",
        "# Feature that can be grouped into buckets\n",
        "BUCKET_FEATURE_DICT = {'age': 4}\n",
        "\n",
        "# Number of out-of-vocabulary buckets\n",
        "NUM_OOV_BUCKETS = 5\n",
        "\n",
        "# Feature that the model will predict\n",
        "LABEL_KEY = 'label'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4AJ9hBs94YJm"
      },
      "source": [
        "# Set the transform module filename\n",
        "_census_transform_module_file = 'census_transform.py'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MYmxxx9A4YJn"
      },
      "source": [
        "%%writefile {_census_transform_module_file}\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_transform as tft\n",
        "\n",
        "# import constants from cells above\n",
        "import census_constants\n",
        "\n",
        "# Unpack the contents of the constants module\n",
        "_NUMERIC_FEATURE_KEYS = census_constants.NUMERIC_FEATURE_KEYS\n",
        "_VOCAB_FEATURE_DICT = census_constants.VOCAB_FEATURE_DICT\n",
        "_BUCKET_FEATURE_DICT = census_constants.BUCKET_FEATURE_DICT\n",
        "_NUM_OOV_BUCKETS = census_constants.NUM_OOV_BUCKETS\n",
        "_LABEL_KEY = census_constants.LABEL_KEY\n",
        "\n",
        "# Define the transformations\n",
        "def preprocessing_fn(inputs):\n",
        "    \"\"\"tf.transform's callback function for preprocessing inputs.\n",
        "    Args:\n",
        "        inputs: map from feature keys to raw not-yet-transformed features.\n",
        "    Returns:\n",
        "        Map from string feature key to transformed feature operations.\n",
        "    \"\"\"\n",
        "\n",
        "    # Initialize outputs dictionary\n",
        "    outputs = {}\n",
        "\n",
        "    # Scale these features to the range [0,1]\n",
        "    for key in _NUMERIC_FEATURE_KEYS:\n",
        "        scaled = tft.scale_to_0_1(inputs[key])\n",
        "        outputs[key] = tf.reshape(scaled, [-1])\n",
        "\n",
        "    # Convert strings to indices and convert to one-hot vectors\n",
        "    for key, vocab_size in _VOCAB_FEATURE_DICT.items():\n",
        "        indices = tft.compute_and_apply_vocabulary(inputs[key], num_oov_buckets=_NUM_OOV_BUCKETS)\n",
        "        one_hot = tf.one_hot(indices, vocab_size + _NUM_OOV_BUCKETS)\n",
        "        outputs[key] = tf.reshape(one_hot, [-1, vocab_size + _NUM_OOV_BUCKETS])\n",
        "\n",
        "    # Bucketize this feature and convert to one-hot vectors\n",
        "    for key, num_buckets in _BUCKET_FEATURE_DICT.items():\n",
        "        indices = tft.bucketize(inputs[key], num_buckets)\n",
        "        one_hot = tf.one_hot(indices, num_buckets)\n",
        "        outputs[key] = tf.reshape(one_hot, [-1, num_buckets])\n",
        "\n",
        "    # Cast label to float\n",
        "    outputs[_LABEL_KEY] = tf.cast(inputs[_LABEL_KEY], tf.float32)\n",
        "\n",
        "    return outputs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wgbmZr3sgbWW"
      },
      "source": [
        "Now, we pass in this feature engineering code to the `Transform` component and run it to transform your data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jHfhth_GiZI9"
      },
      "source": [
        "# Run the Transform component\n",
        "transform = tfx.components.Transform(\n",
        "    examples=example_gen.outputs['examples'],\n",
        "    schema=schema_gen.outputs['schema'],\n",
        "    module_file=os.path.abspath(_census_transform_module_file))\n",
        "context.run(transform, enable_cache=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LIwwKRmhc2Ve"
      },
      "source": [
        "You can see a sample result for one row with the code below. Notice that the numeric features are indeed scaled and the categorical features are now one-hot encoded."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pwbW2zPKR_S4"
      },
      "source": [
        "# Get the URI of the output artifact representing the transformed examples\n",
        "train_uri = os.path.join(transform.outputs['transformed_examples'].get()[0].uri, 'Split-train')\n",
        "\n",
        "# Get the list of files in this directory (all compressed TFRecord files)\n",
        "tfrecord_filenames = [os.path.join(train_uri, name)\n",
        "                      for name in os.listdir(train_uri)]\n",
        "\n",
        "# Create a `TFRecordDataset` to read these files\n",
        "dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type=\"GZIP\")\n",
        "\n",
        "# Decode the first record and print output\n",
        "for tfrecord in dataset.take(1):\n",
        "  serialized_example = tfrecord.numpy()\n",
        "  example = tf.train.Example()\n",
        "  example.ParseFromString(serialized_example)\n",
        "  pp.pprint(example)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyFkBd9AR1sy"
      },
      "source": [
        "As you already know, the `Transform` component not only outputs the transformed examples but also the transformation graph. This should be used on all inputs when your model is deployed to ensure that it is transformed the same way as your training data. Else, it can produce training-serving skew which leads to noisy predictions.\n",
        "\n",
        "The `Transform` component stores related files in its `transform_graph` output and it would be good to quickly review its contents before we move on to the next component. As shown below, the URI of this output points to a directory containing three subdirectories."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5tRw4DneR3i7"
      },
      "source": [
        "# Get URI and list subdirectories\n",
        "graph_uri = transform.outputs['transform_graph'].get()[0].uri\n",
        "os.listdir(graph_uri)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fqV54CIR6Pu"
      },
      "source": [
        "* The `transformed_metadata` subdirectory contains the schema of the preprocessed data. \n",
        "* The `transform_fn` subdirectory contains the actual preprocessing graph. \n",
        "* The `metadata` subdirectory contains the schema of the original data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OBJFtnl6lCg9"
      },
      "source": [
        "### Trainer\n",
        "\n",
        "Next, you will now build the model to make your predictions. As mentioned earlier, this is a binary classifier where the label is 1 if a person earns more than 50k USD and 0 if less than or equal. The model used here uses the [wide and deep model](https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html) as reference but feel free to modify after you've completed the exercise. Also for simplicity, the hyperparameters (e.g. number of hidden units) have been hardcoded but feel free to use a `Tuner` component as you did in Week 1 if you want to get some practice.\n",
        "\n",
        "As a reminder, it is best to start from `run_fn()` when you're reviewing the module file below. The `Trainer` component looks for that function first. All other functions defined in the module are just utility functions for `run_fn()`.\n",
        "\n",
        "Another thing you will notice below is the `_get_tf_serving_examples_signature()` function. This is tied to the `serving_default` [signature](https://www.tensorflow.org/guide/saved_model#specifying_signatures_during_export) which makes it possible for you to just pass in raw features when the model is served for inference. You have seen this in action in the previous lab. This is done by decorating the enclosing function, `serve_tf_examples_fn()`, with [tf.function](https://www.tensorflow.org/guide/intro_to_graphs). This indicates that the computation will be done by first tracing a [Tensorflow graph](https://www.tensorflow.org/api_docs/python/tf/Graph). You will notice that this function uses `model.tft_layer_inference` which comes from `transform_graph` output. Now when you call the `.get_concrete_function()` method on this tf.function in `run_fn()`, you are creating the graph that will be used in later computations. This graph is used whenever you pass in an `examples` argument pointing to a Tensor with `tf.string` dtype. That matches the format of the serialized batches of data you used in the previous lab.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N1376oq04YJt"
      },
      "source": [
        "# Declare trainer module file\n",
        "_census_trainer_module_file = 'census_trainer.py'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nf9UuNng4YJu"
      },
      "source": [
        "%%writefile {_census_trainer_module_file}\n",
        "\n",
        "from typing import List, Text\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_transform as tft\n",
        "from tensorflow_transform.tf_metadata import schema_utils\n",
        "\n",
        "from tfx.components.trainer.fn_args_utils import DataAccessor, FnArgs\n",
        "from tfx_bsl.public.tfxio import TensorFlowDatasetOptions\n",
        "\n",
        "# import same constants from transform module\n",
        "import census_constants\n",
        "\n",
        "# Unpack the contents of the constants module\n",
        "_NUMERIC_FEATURE_KEYS = census_constants.NUMERIC_FEATURE_KEYS\n",
        "_VOCAB_FEATURE_DICT = census_constants.VOCAB_FEATURE_DICT\n",
        "_BUCKET_FEATURE_DICT = census_constants.BUCKET_FEATURE_DICT\n",
        "_NUM_OOV_BUCKETS = census_constants.NUM_OOV_BUCKETS\n",
        "_LABEL_KEY = census_constants.LABEL_KEY\n",
        "\n",
        "\n",
        "def _gzip_reader_fn(filenames):\n",
        "  '''Load compressed dataset\n",
        "  \n",
        "  Args:\n",
        "    filenames - filenames of TFRecords to load\n",
        "\n",
        "  Returns:\n",
        "    TFRecordDataset loaded from the filenames\n",
        "  '''\n",
        "\n",
        "  # Load the dataset. Specify the compression type since it is saved as `.gz`\n",
        "  return tf.data.TFRecordDataset(filenames, compression_type='GZIP')\n",
        "\n",
        "\n",
        "def _input_fn(file_pattern,\n",
        "              tf_transform_output,\n",
        "              num_epochs=None,\n",
        "              batch_size=32) -> tf.data.Dataset:\n",
        "  '''Create batches of features and labels from TF Records\n",
        "\n",
        "  Args:\n",
        "    file_pattern - List of files or patterns of file paths containing Example records.\n",
        "    tf_transform_output - transform output graph\n",
        "    num_epochs - Integer specifying the number of times to read through the dataset. \n",
        "            If None, cycles through the dataset forever.\n",
        "    batch_size - An int representing the number of records to combine in a single batch.\n",
        "\n",
        "  Returns:\n",
        "    A dataset of dict elements, (or a tuple of dict elements and label). \n",
        "    Each dict maps feature keys to Tensor or SparseTensor objects.\n",
        "  '''\n",
        "\n",
        "  # Get post-transfrom feature spec\n",
        "  transformed_feature_spec = (\n",
        "      tf_transform_output.transformed_feature_spec().copy())\n",
        "  \n",
        "  # Create batches of data\n",
        "  dataset = tf.data.experimental.make_batched_features_dataset(\n",
        "      file_pattern=file_pattern,\n",
        "      batch_size=batch_size,\n",
        "      features=transformed_feature_spec,\n",
        "      reader=_gzip_reader_fn,\n",
        "      num_epochs=num_epochs,\n",
        "      label_key=_LABEL_KEY\n",
        "      )\n",
        "  \n",
        "  return dataset\n",
        "\n",
        "\n",
        "def _get_tf_examples_serving_signature(model, tf_transform_output):\n",
        "  \"\"\"Returns a serving signature that accepts `tensorflow.Example`.\"\"\"\n",
        "\n",
        "  # Get the transformation graph\n",
        "  model.tft_layer_inference = tf_transform_output.transform_features_layer()\n",
        "\n",
        "  @tf.function(input_signature=[\n",
        "      tf.TensorSpec(shape=[None], dtype=tf.string, name='examples')\n",
        "  ])\n",
        "  def serve_tf_examples_fn(serialized_tf_example):\n",
        "    \"\"\"Returns the output to be used in the serving signature.\"\"\"\n",
        "    \n",
        "    # Get pre-transform feature spec\n",
        "    raw_feature_spec = tf_transform_output.raw_feature_spec()\n",
        "    \n",
        "    # Remove label feature since these will not be present at serving time.\n",
        "    raw_feature_spec.pop(_LABEL_KEY)\n",
        "    \n",
        "    # Parse raw examples into a dictionary of tensors matching the feature spec\n",
        "    raw_features = tf.io.parse_example(serialized_tf_example, raw_feature_spec)\n",
        "    \n",
        "    # Transform the raw examples using the transform graph\n",
        "    transformed_features = model.tft_layer_inference(raw_features)\n",
        "\n",
        "    # Get predictions using the transformed features\n",
        "    outputs = model(transformed_features)\n",
        "\n",
        "    return {'outputs': outputs}\n",
        "\n",
        "  return serve_tf_examples_fn\n",
        "\n",
        "\n",
        "def export_serving_model(tf_transform_output, model, output_dir):\n",
        "  \"\"\"Exports a keras model for serving.\n",
        "  Args:\n",
        "    tf_transform_output: Wrapper around output of tf.Transform.\n",
        "    model: A keras model to export for serving.\n",
        "    output_dir: A directory where the model will be exported to.\n",
        "  \"\"\"\n",
        "  # The layer has to be saved to the model for keras tracking purpases.\n",
        "  model.tft_layer = tf_transform_output.transform_features_layer()\n",
        "\n",
        "  signatures = {\n",
        "      'serving_default':\n",
        "          _get_tf_examples_serving_signature(model, tf_transform_output)\n",
        "  }\n",
        "\n",
        "  model.save(output_dir, save_format='tf', signatures=signatures)\n",
        "\n",
        "\n",
        "def _build_keras_model(hidden_units: List[int] = None) -> tf.keras.Model:\n",
        "  \"\"\"Creates a DNN Keras model for classifying income data.\n",
        "\n",
        "  Args:\n",
        "    hidden_units: [int], the layer sizes of the DNN (input layer first).\n",
        "\n",
        "  Returns:\n",
        "    A keras Model.\n",
        "  \"\"\"\n",
        "\n",
        "  # Use helper function to create the model\n",
        "  model = _wide_and_deep_classifier(\n",
        "      dnn_hidden_units=hidden_units or [100, 70, 50, 25])\n",
        "  \n",
        "  return model\n",
        "\n",
        "\n",
        "def _wide_and_deep_classifier(dnn_hidden_units):\n",
        "  \"\"\"Build a simple keras wide and deep model using the Functional API.\n",
        "\n",
        "  Args:\n",
        "    wide_columns: Feature columns wrapped in indicator_column for wide (linear)\n",
        "      part of the model.\n",
        "    deep_columns: Feature columns for deep part of the model.\n",
        "    dnn_hidden_units: [int], the layer sizes of the hidden DNN.\n",
        "\n",
        "  Returns:\n",
        "    A Wide and Deep Keras model\n",
        "  \"\"\"\n",
        "\n",
        "  # Define input layers for numeric keys\n",
        "  input_numeric = [\n",
        "      tf.keras.layers.Input(name=colname, shape=(1,), dtype=tf.float32)\n",
        "      for colname in _NUMERIC_FEATURE_KEYS\n",
        "  ]\n",
        "\n",
        "  # Define input layers for vocab keys\n",
        "  input_categorical = [\n",
        "      tf.keras.layers.Input(name=colname, shape=(vocab_size + _NUM_OOV_BUCKETS,), dtype=tf.float32)\n",
        "      for colname, vocab_size in _VOCAB_FEATURE_DICT.items()\n",
        "  ]\n",
        "\n",
        "  # Define input layers for bucket key\n",
        "  input_categorical += [\n",
        "      tf.keras.layers.Input(name=colname, shape=(num_buckets,), dtype=tf.float32)\n",
        "      for colname, num_buckets in _BUCKET_FEATURE_DICT.items()\n",
        "  ]\n",
        "\n",
        "  # Concatenate numeric inputs\n",
        "  deep = tf.keras.layers.concatenate(input_numeric)\n",
        "\n",
        "  # Create deep dense network for numeric inputs\n",
        "  for numnodes in dnn_hidden_units:\n",
        "    deep = tf.keras.layers.Dense(numnodes)(deep)\n",
        "\n",
        "  # Concatenate categorical inputs\n",
        "  wide = tf.keras.layers.concatenate(input_categorical)\n",
        "\n",
        "  # Create shallow dense network for categorical inputs\n",
        "  wide = tf.keras.layers.Dense(128, activation='relu')(wide)\n",
        "\n",
        "  # Combine wide and deep networks\n",
        "  combined = tf.keras.layers.concatenate([deep, wide])\n",
        "                                              \n",
        "  # Define output of binary classifier\n",
        "  output = tf.keras.layers.Dense(\n",
        "      1, activation='sigmoid')(combined)\n",
        "\n",
        "  # Setup combined input\n",
        "  input_layers = input_numeric + input_categorical\n",
        "\n",
        "  # Create the Keras model\n",
        "  model = tf.keras.Model(input_layers, output)\n",
        "\n",
        "  # Define training parameters\n",
        "  model.compile(\n",
        "      loss='binary_crossentropy',\n",
        "      optimizer=tf.keras.optimizers.Adam(lr=0.001),\n",
        "      metrics='binary_accuracy')\n",
        "  \n",
        "  # Print model summary\n",
        "  model.summary()\n",
        "\n",
        "  return model\n",
        "\n",
        "\n",
        "# TFX Trainer will call this function.\n",
        "def run_fn(fn_args: FnArgs):\n",
        "  \"\"\"Defines and trains the model.\n",
        "  \n",
        "  Args:\n",
        "    fn_args: Holds args as name/value pairs. Refer here for the complete attributes: \n",
        "    https://www.tensorflow.org/tfx/api_docs/python/tfx/components/trainer/fn_args_utils/FnArgs#attributes\n",
        "  \"\"\"\n",
        "\n",
        "  # Number of nodes in the first layer of the DNN\n",
        "  first_dnn_layer_size = 100\n",
        "  num_dnn_layers = 4\n",
        "  dnn_decay_factor = 0.7\n",
        "\n",
        "  # Get transform output (i.e. transform graph) wrapper\n",
        "  tf_transform_output = tft.TFTransformOutput(fn_args.transform_output)\n",
        "\n",
        "  # Create batches of train and eval sets\n",
        "  train_dataset = _input_fn(fn_args.train_files[0], tf_transform_output, 10)\n",
        "  eval_dataset = _input_fn(fn_args.eval_files[0], tf_transform_output, 10)\n",
        "\n",
        "  # Build the model\n",
        "  model = _build_keras_model(\n",
        "      # Construct layers sizes with exponential decay\n",
        "      hidden_units=[\n",
        "          max(2, int(first_dnn_layer_size * dnn_decay_factor**i))\n",
        "          for i in range(num_dnn_layers)\n",
        "      ])\n",
        "  \n",
        "  # Callback for TensorBoard\n",
        "  tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
        "      log_dir=fn_args.model_run_dir, update_freq='batch')\n",
        "\n",
        "\n",
        "  # Train the model\n",
        "  model.fit(\n",
        "      train_dataset,\n",
        "      steps_per_epoch=fn_args.train_steps,\n",
        "      validation_data=eval_dataset,\n",
        "      validation_steps=fn_args.eval_steps,\n",
        "      callbacks=[tensorboard_callback])\n",
        "  \n",
        "\n",
        "  # Export the model.\n",
        "  export_serving_model(tf_transform_output, model, fn_args.serving_model_dir)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GY4yTRaX4YJx"
      },
      "source": [
        "Now, we pass in this model code to the `Trainer` component and run it to train the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "429-vvCWibO0"
      },
      "source": [
        "trainer = tfx.components.Trainer(\n",
        "    module_file=os.path.abspath(_census_trainer_module_file),\n",
        "    examples=transform.outputs['transformed_examples'],\n",
        "    transform_graph=transform.outputs['transform_graph'],\n",
        "    schema=schema_gen.outputs['schema'],\n",
        "    train_args=tfx.proto.TrainArgs(num_steps=50),\n",
        "    eval_args=tfx.proto.EvalArgs(num_steps=50))\n",
        "context.run(trainer, enable_cache=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Cql1G35StJp"
      },
      "source": [
        "Let's review the outputs of this component. The `model` output points to the model output itself."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bXe62WE0S0Ek"
      },
      "source": [
        "# Get `model` output of the component\n",
        "model_artifact_dir = trainer.outputs['model'].get()[0].uri\n",
        "\n",
        "# List top-level directory\n",
        "pp.pprint(os.listdir(model_artifact_dir))\n",
        "\n",
        "# Get model directory\n",
        "model_dir = os.path.join(model_artifact_dir, 'Format-Serving')\n",
        "\n",
        "# List subdirectories\n",
        "pp.pprint(os.listdir(model_dir))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DfjOmSro6Q3Y"
      },
      "source": [
        "The `model_run` output acts as the working directory and can be used to output non-model related output (e.g., TensorBoard logs)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-APzqz2NeAyj"
      },
      "source": [
        "# Get `model_run` output URI\n",
        "model_run_artifact_dir = trainer.outputs['model_run'].get()[0].uri\n",
        "\n",
        "# Load results to Tensorboard\n",
        "%load_ext tensorboard\n",
        "%tensorboard --logdir {model_run_artifact_dir}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmPftrv0lEQy"
      },
      "source": [
        "### Evaluator\n",
        "\n",
        "The `Evaluator` component computes model performance metrics over the evaluation set using the [TensorFlow Model Analysis](https://www.tensorflow.org/tfx/model_analysis/get_started) library. The `Evaluator` can also optionally validate that a newly trained model is better than the previous model. This is useful in a production pipeline setting where you may automatically train and validate a model every day.\n",
        "\n",
        "There's a few steps needed to setup this component and you will do it in the next cells."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RfNYwkKEQIuE"
      },
      "source": [
        "#### Define EvalConfig\n",
        "\n",
        "First, you will define the `EvalConfig` as you did in the previous lab. You can also set thresholds so you can compare subsequent models to it. The module below should look familiar. One minor difference is you don't have to define the candidate and baseline model names in the `model_specs`. That is automatically detected."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fVhfzzh9PDEx"
      },
      "source": [
        "eval_config = tfma.EvalConfig(\n",
        "    model_specs=[\n",
        "        # This assumes a serving model with signature 'serving_default'.\n",
        "        tfma.ModelSpec(\n",
        "            signature_name='serving_default',\n",
        "            label_key='label'\n",
        "            )\n",
        "        ],\n",
        "    metrics_specs=[\n",
        "        tfma.MetricsSpec(\n",
        "            # The metrics added here are in addition to those saved with the\n",
        "            # model (assuming either a keras model or EvalSavedModel is used).\n",
        "            # Any metrics added into the saved model (for example using\n",
        "            # model.compile(..., metrics=[...]), etc) will be computed\n",
        "            # automatically.\n",
        "            # To add validation thresholds for metrics saved with the model,\n",
        "            # add them keyed by metric name to the thresholds map.\n",
        "            metrics=[\n",
        "                tfma.MetricConfig(class_name='ExampleCount'),\n",
        "                tfma.MetricConfig(class_name='BinaryAccuracy',\n",
        "                      threshold=tfma.MetricThreshold(\n",
        "                          value_threshold=tfma.GenericValueThreshold(\n",
        "                              lower_bound={'value': 0.5}),\n",
        "                              # Change threshold will be ignored if there is no\n",
        "                              # baseline model resolved from MLMD (first run).\n",
        "                              change_threshold=tfma.GenericChangeThreshold(\n",
        "                                  direction=tfma.MetricDirection.HIGHER_IS_BETTER,\n",
        "                                  absolute={'value': -1e-10}\n",
        "                                  )\n",
        "                              )\n",
        "                      )\n",
        "            ]\n",
        "        )\n",
        "    ],\n",
        "    slicing_specs=[\n",
        "        # An empty slice spec means the overall slice, i.e. the whole dataset.\n",
        "        tfma.SlicingSpec(),\n",
        "        # Data can be sliced along a feature column.\n",
        "        tfma.SlicingSpec(\n",
        "            feature_keys=['sex']),\n",
        "        tfma.SlicingSpec(\n",
        "            feature_keys=['race']),\n",
        "    ])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9mBdKH1F8JuT"
      },
      "source": [
        "#### Resolve latest blessed model\n",
        "\n",
        "If you remember in the last lab, you were able to validate a candidate model against a baseline by modifying the `EvalConfig` and `EvalSharedModel` definitions. That is also possible using the `Evaluator` component and you will see how it is done in this section.\n",
        "\n",
        "First thing to note is that the `Evaluator` marks a model as `BLESSED` if it meets the metrics thresholds you set in the eval config module. You can load the latest blessed model by using the [`LatestBlessedModelStrategy`](https://www.tensorflow.org/tfx/api_docs/python/tfx/dsl/input_resolution/strategies/latest_blessed_model_strategy/LatestBlessedModelStrategy) with the [Resolver](https://www.tensorflow.org/tfx/api_docs/python/tfx/dsl/components/common/resolver/Resolver) component. This component takes care of finding the latest blessed model for you so you don't have to remember it manually. The syntax is shown below. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lkujxtpUccUl"
      },
      "source": [
        "# Setup the Resolver node to find the latest blessed model\n",
        "model_resolver = tfx.dsl.Resolver(\n",
        "      strategy_class=tfx.dsl.experimental.LatestBlessedModelStrategy,\n",
        "      model=tfx.dsl.Channel(type=tfx.types.standard_artifacts.Model),\n",
        "      model_blessing=tfx.dsl.Channel(\n",
        "          type=tfx.types.standard_artifacts.ModelBlessing)).with_id(\n",
        "              'latest_blessed_model_resolver')\n",
        "\n",
        "# Run the Resolver node\n",
        "context.run(model_resolver)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkfsYyb45aSe"
      },
      "source": [
        "As expected, the search yielded 0 artifacts because you haven't evaluated any models yet. You will run this component again in later parts of this notebook and you will see a different result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lw3GL7mw1ynx"
      },
      "source": [
        "# Load Resolver outputs\n",
        "model_resolver.outputs['model']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p_V3bV6i588X"
      },
      "source": [
        "#### Run the Evaluator component\n",
        "\n",
        "With the `EvalConfig` defined and code to load the latest blessed model available, you can proceed to run the `Evaluator` component. \n",
        "\n",
        "You will notice that two models are passed into the component. The `Trainer` output will serve as the candidate model while the output of the `Resolver` will be the baseline model. While you can indeed run the `Evaluator` without comparing two models, it would likely be required in production environments so it's best to include it. Since the `Resolver` doesn't have any results yet, the `Evaluator` will just mark the candidate model as `BLESSED` in this run.\n",
        "\n",
        "Aside from the eval config and models (i.e. Trainer and Resolver output), you will also pass in the *raw* examples from `ExampleGen`. By default, the component will look for the `eval` split of these examples and since you've defined the serving signature, these will be transformed internally before feeding to the model inputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zjcx8g6mihSt"
      },
      "source": [
        "# Setup and run the Evaluator component\n",
        "evaluator = tfx.components.Evaluator(\n",
        "    examples=example_gen.outputs['examples'],\n",
        "    model=trainer.outputs['model'],\n",
        "    baseline_model=model_resolver.outputs['model'],\n",
        "    eval_config=eval_config)\n",
        "context.run(evaluator, enable_cache=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AeCVkBusS_8g"
      },
      "source": [
        "Now let's examine the output artifacts of `Evaluator`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CA2WpWHbn3mn"
      },
      "source": [
        "# Print component output keys\n",
        "evaluator.outputs.keys()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-D1ITe89-2J"
      },
      "source": [
        "The `blessing` output simply states if the candidate model was blessed. The artifact URI will have a `BLESSED` or `NOT_BLESSED` file depending on the result. As mentioned earlier, this first run will pass the evaluation because there is no baseline model yet."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XV8mypPX_Jpv"
      },
      "source": [
        "# Get `Evaluator` blessing output URI\n",
        "blessing_uri = evaluator.outputs['blessing'].get()[0].uri\n",
        "\n",
        "# List files under URI\n",
        "os.listdir(blessing_uri)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y5TMskWe9LL0"
      },
      "source": [
        "The `evaluation` output, on the other hand, contains the evaluation logs and can be used to visualize the global metrics on the entire evaluation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U729j5X5QQUQ"
      },
      "source": [
        "# Visualize the evaluation results\n",
        "context.show(evaluator.outputs['evaluation'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t-tI4p6m-OAn"
      },
      "source": [
        "To see the individual slices, you will need to import TFMA and use the commands you learned in the previous lab."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pyis6iy0HLdi"
      },
      "source": [
        "import tensorflow_model_analysis as tfma\n",
        "\n",
        "# Get the TFMA output result path and load the result.\n",
        "PATH_TO_RESULT = evaluator.outputs['evaluation'].get()[0].uri\n",
        "tfma_result = tfma.load_eval_result(PATH_TO_RESULT)\n",
        "\n",
        "# Show data sliced along a feature column\n",
        "tfma.view.render_slicing_metrics(\n",
        "    tfma_result, slicing_column='sex')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hM1tFkOVSBa0"
      },
      "source": [
        "You can also use TFMA to load the validation results as before by specifying the output URI of the evaluation output. This would be more useful if your model was not blessed because you can see the metric failure prompts. Try to simulate this later by training with fewer epochs (or raising the threshold) and see the results you get here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lxa5G08bSJ8a"
      },
      "source": [
        "# Get `evaluation` output URI\n",
        "PATH_TO_RESULT = evaluator.outputs['evaluation'].get()[0].uri\n",
        "\n",
        "# Print validation result\n",
        "print(tfma.load_validation_result(PATH_TO_RESULT))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QWieYxRJBtI9"
      },
      "source": [
        "Now that your `Evaluator` has finished running, the `Resolver` component should be able to detect the latest blessed model. Let's run the component again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NZdLcnEc3G23"
      },
      "source": [
        "# Re-run the Resolver component\n",
        "context.run(model_resolver)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4YMkNm72B8y6"
      },
      "source": [
        "You should now see an artifact in the component outputs. You can also get it programmatically as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XJ5o0z353OC1"
      },
      "source": [
        "# Get path to latest blessed model\n",
        "model_resolver.outputs['model'].get()[0].uri"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nrsxo8KWCQ6m"
      },
      "source": [
        "#### Comparing two models\n",
        "\n",
        "Now let's see how `Evaluator` compares two models. You will train the same model with more epochs and this should hopefully result in higher accuracy and overall metrics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d1LqkOQ03dD9"
      },
      "source": [
        "# Setup trainer to train with more epochs\n",
        "trainer = tfx.components.Trainer(\n",
        "    module_file=os.path.abspath(_census_trainer_module_file),\n",
        "    examples=transform.outputs['transformed_examples'],\n",
        "    transform_graph=transform.outputs['transform_graph'],\n",
        "    schema=schema_gen.outputs['schema'],\n",
        "    train_args=tfx.proto.TrainArgs(num_steps=500),\n",
        "    eval_args=tfx.proto.EvalArgs(num_steps=200))\n",
        "\n",
        "# Run trainer\n",
        "context.run(trainer, enable_cache=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RURw636C09I"
      },
      "source": [
        "You will re-run the evaluator but you will specify the latest trainer output as the candidate model. The baseline is automatically found with the Resolver node."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GR6mIb4A3PFE"
      },
      "source": [
        "# Setup and run the Evaluator component\n",
        "evaluator = tfx.components.Evaluator(\n",
        "    examples=example_gen.outputs['examples'],\n",
        "    model=trainer.outputs['model'],\n",
        "    baseline_model=model_resolver.outputs['model'],\n",
        "    eval_config=eval_config)\n",
        "context.run(evaluator, enable_cache=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_V1xETMDfkE"
      },
      "source": [
        "Depending on the result, the Resolver should reflect the latest blessed model. Since you trained with more epochs, it is most likely that your candidate model will pass the thresholds you set in the eval config. If so, the artifact URI should be different here compared to your earlier runs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBupRi933vIr"
      },
      "source": [
        "# Re-run the resolver\n",
        "context.run(model_resolver, enable_cache=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3rE9G07WrLR6"
      },
      "source": [
        "# Get path to latest blessed model\n",
        "model_resolver.outputs['model'].get()[0].uri"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qU344SYKD999"
      },
      "source": [
        "Finally, the `evaluation` output of the `Evaluator` component will now be able to produce the `diff` results you saw in the previous lab. This will signify if the metrics from the candidate model has indeed improved compared to the baseline. Unlike when using TFMA as a standalone library, visualizing this will just show the results for the candidate (i.e. only one row instead of two in the tabular output in the graph below).\n",
        "\n",
        "*Note: You can ignore the warning about failing to find plots.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wY-99Y9ptfhP"
      },
      "source": [
        "context.show(evaluator.outputs['evaluation'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UR-sivmME-b5"
      },
      "source": [
        "**Congratulations! You can now successfully evaluate your models in a TFX pipeline! This is a critical part of production ML because you want to make sure that subsequent deployments are indeed improving your results. Moreover, you can extract the evaluation results from your pipeline directory for further investigation with TFMA. In the next sections, you will continue to study techniques related to model evaluation and ensuring fairness.**"
      ]
    }
  ]
}